Triangle in a Square

Worked solution

The answer is 11/144 ≈ 0.0764. Sort the points by x-coordinate: x₁ < x₂ < x₃. The triangle's area equals (1/2)(x₃ − x₁)·h, where h is the vertical distance from the middle point to the segment joining the outer two, measured along the line x = x₂: h = |y₂ − (1 − λ)y₁ − λy₃| with λ = (x₂ − x₁)/(x₃ − x₁). Given the extreme x-values, x₂ is uniform between them, so λ is uniform on [0, 1] and independent of the width w = x₃ − x₁; also the y's are independent uniforms unaffected by sorting on x. The range of three uniforms has E[w] = 1/2. A direct polynomial integration gives E|y₂ − (1 − λ)y₁ − λy₃| averaged over λ equal to 11/36. Hence E[area] = (1/2)·(1/2)·(11/36) = 11/144. As a consequence (by Sylvester's four-point argument), four random points in a square form a convex quadrilateral with probability 1 − 4·11/144 = 25/36.