We have a n-sided convex polygon P. How many r-sided polygons (r<n), with its vertices among those of P, can be formed such that it has no sides (edges) in common with P?

Explanation:
Name the vertices $a_1,a_2,\dots <!--\; \dots \; -->,a_n$. Then focus on the gaps. The number of gaps is $(r-<!--\; -\; -->1)$ (say, $g_1+g_2+\cdots <!--\; \cdots \; -->+g(r-<!--\; -\; -->1))$. Then use these gaps as variables of a multinomial. The sum of these gaps ranges from $r-<!--\; -\; -->1$ to $n-<!--\; -\; -->r-<!--\; -\; -->1$. Now use the multinomial theorem, and/or Stars and bars theorem 1. (There has to be at least one gap between the vertices). Now you will get a sum of binomial coefficients which can be solved easily using the Hockey stick identity.

Step 1
Label the vertices of P, in order, $1,\dots <!--\; \dots \; -->,n$ starting at any vertex you like.
An r-sided polygon in P then corresponds to an increasing sequence of r indices between 1 and n. For $n=5$, for instance, you might take 1,3,5, meaning that your "internal" polygon has vertices 1, 3, and 5. The only tricky thing in your case is that no two adjacent numbers can be picked.
[Reason: any r-sided polygon has a lowest index. If your indexes don't increase, then you get a crossing (why?). If adjacent numbers are picked, you get an edge of P.]
Thus you need to count how many length-r subsequences 1,2,…,n has, where (a) no two elements of the sequence are adjacent, and (b) you don't pick both "1" and n to be in the subsequence.
So: pick a starting vertex, and then pick a "gap" between adjacent vertices. In fact, pick $r-<!--\; -\; -->1$ numbers. So a starting vertex of 2 and the sequence (1,1,2) in a 10-sided polygon would define the subpolygon
(2, 4, 6, 9)
with gaps of size 1, 1, and 2. (A 'gap' is how many vertices you skipped over.)
Step 2
The only problem is that you might start with vertex 1 and end with vertex n, and thus include the edge n,1. So let's pick r gaps instead. (In the example above, they are (1,1,2,3)). These must have the properties that
1. They're positive
2. They sum to $n-<!--\; -\; -->r+1$
3. The sum of all except that last one, plus r, must be no more than n
The "last edge" problem makes this ugly. So let's split into two subproblems:
1. How many internal polygons are there that don't contain vertex 1?
2. How many are there that DO contain vertex 1?
For part 1, in a 10-vertex polygon, if your internal 3-vertex polygon has a list of verts like (2, 5, 7), you can imagine that you start counting "gaps" at a fictitious vertex 0, and you get gaps of 1,2,1,3 for this sequence. The sum of the gaps is $10-<!--\; -\; -->3$. In short:
The answer to part 1 is a "stars and bars" problem for n stars and bars among which are $r+1$ stars. (stars correspond to vertices)