Convexity is a geometric concept that extends beyond polygons. By definition, a subset of the plane is convex if, for every pair of points chosen from it, the line segment joining them lies entirely within the subset.
To apply this definition to polygons, we must consider the polygon's interior. So it must have an interior! This is why we consider only simple polygons (see the article What Is a Polygon?). A self-intersecting polygon is therefore necessarily non-convex.
A theorem will help us here. If, for each of its sides, a polygon lies entirely within one of the half-planes bounded by the line containing that side, then it is convex.
Another theorem tells us that a polygon is convex if all its interior angles are less than π. Thus, a single interior angle greater than π is enough for the polygon not to be convex.