Your olives are infested with larvae (maggots) of the olive fly. This fruit fly pest is native to the Mediterranean region. It was discovered in California in 1996 infesting olives on landscape trees in Los Angeles County and has since spread through much of the state.
The olive fly is a major pest of olives. The larvae feed on olives and cause early fruit drop. Infested fruit are not suitable for pickling. Larval feeding in the fruit and secondary bacterial decay also degrade the quality of oil obtained from infested fruit.
The adult olive fly is about 3/16 of an inch long and has a brown body with a yellowish to light brown head. The wings are clear with dark veins and a small black spot at the wing tip. The female flies begin laying eggs about August when fruit begins to soften. They pierce the skin of an olive and deposit a single egg in the pulp. Eggs hatch into yellowish-white maggots that feed throughout the olive. They grow to a length of about 1/5 of an inch then enter a resting stage and pupate just beneath the skin. Adult flies emerge in as little as 30 days during warm weather. Subsequent generations follow the same pattern until fall when larvae abandon the fruit to pupate in the soil. Adult flies may not emerge from over-wintering pupae for five to six months. In mild climates adult flies may also over-winter near olive trees.
The olive fly is difficult to control in home gardens. Adult flies must be killed before they lay eggs in fruit. The eggs and larvae in fruit are protected and contact insecticides will not kill them. Traps can be used to determine when adult flies are active so an insecticide and bait spray can be applied at the proper time. Traps alone will not catch enough flies to protect fruit from damage. Olive trees on a residential property can be treated for olive fly. However this may not be effective if other olive trees located nearby are not treated.
For more information see UC Pest Note 74112 Olive Fruit Fly http://www.ipm.ucdavis.edu/PMG/PESTNOTES/pn74112.html
By V. Lazaneo Urban Horticulture Advisor, Emeritus, UC Cooperative Extension, August 2012