Q5 is true if (as you assumed), the space of lotteries is the space of distributions over a finite set. (For a general convex set, you can get long-line phenomena.)
First, without proof, I'll state the following generalization.
Theorem 1. Let be a relation on a convex space satisfying axioms A1, A2, A3, and the following additional continuity axiom. For all , the set
is open in . Then, there exists a function from to the long line such that iff .
The proof is not too different, but simpler, if we also assume A4. In particular, we no longer need the extra continuity axiom, and we get a stronger conclusion. Nate sketched part of the proof of this already, but I want to be clearer about what is stated and skip fewer steps. In particular, I'm not sure how Nate's hypotheses rule out examples that require long-line-valued functions—maybe he's assuming that the domain of the preference relation is a finite-dimensional simplex like I am, but none of his arguments use this explicitly.
Theorem 2. Let be a relation on a finite-dimensional simplex satisfying axioms A1-A4. Then, there is a quasiconcave function such that iff .
First, I'll set up some definitions and a lemma. For any lotteries , , let denote the line segment
We say that preferences are increasing along a line segment if whenever , we have
We will also use open and half-open interval notation in the corresponding way.
Lemma. Let be a preference relation on a finite-dimensional simplex satisfying axioms A1-A4. Then, there are -minimal and -maximal elements in .
Proof. First, we show that there is a minimal element. Axiom A4 states that for any mixture , either or . By induction, it follows more generally that any convex combination C of finitely many elements satisfies for some . But every element is a convex combination of the vertices of , so some vertex of is -minimal.
The proof that there is a maximal element is more complex. Consider the family of sets
This is a prefilter, so since is compact ( here carries the Euclidean metric), it has a cluster point . Either will be a maximal element, or we will find some other maximal element. In particular, take any . We are done if is a maximal element; otherwise, pick . By the construction of , for every , we can pick some within a distance of from B. Now, if we show that itself satisfies , it will follow that is maximal.
The idea is to pass from our sequence , with limit , to another sequence lying on a line segment with endpoint . We can use axiom A4, which is a kind of convexity, to control the preference relation on convex combinations of our points , so these are the points that we will construct along a line segment. Once we have this line segment, we can finish by using A3, which is a kind of continuity restricted to line segments, to control itself.
Let be the set of lotteries in the affine span of the set . Then, if we take some index set such that is a maximal affinely independent tuple, it follows that affinely generates . Hence, the convex combination
i.e. the barycenter of the simplex with vertices at , is in the interior of the convex hull of relative to , so we can pick some such that the -ball around relative to is contained in this simplex.
Now, we will see that every lottery in the set satisfies . For any , pick so that is in the -ball around . Since the tangent vector has length less than , the lottery
is in the -ball around , and it is in , so it is in the simplex with vertices . Then, by A4, and by hypothesis. So, applying A4 again,
Using A4 one more time, it follows that every lottery
satisfies , and hence every lottery .
Now we can finish up. If then, using A3 and the fact that , there would have to be some lottery in that is -equivalent to A, but this would contradict what we just concluded. So, , and so B is -maximal.
Proof of Theorem 2. Let be a -minimal and a -maximal element of . First, we will see that preferences are increasing on , and then we will use this fact to construct a function and show that it has the desired properties. Suppose preferences we not increasing; then, there would be such that is closer to while is closer to , and . Then, would be a convex combination of and , but by the maximality of , contradicting A4.
Now we can construct our utility function using A3; for each -class , we have , so there is some such that
Then, let for all . Since preferences are increasing on , it is immediate that if , then . Conversely, if , we have two cases. If , then , so , and so . Finally, if , then by construction.
Finally, since for all we have iff , it follows immediately that is quasiconcave by A4.