A National Osteoporosis Society survey found a fifth of under-25s are cutting out or reducing dairy in their diet.
It said it was concerned many young adults were putting their health at risk by following eating fads.
Cutting out dairy can be healthy if enough calcium is consumed from other sources, such as nuts, seeds and fish.
The charity surveyed 2,000 adults, including 239 under the age of 25 and 339 aged 25-35.
Milk and dairy food, such as cheese and yoghurt, are important sources of calcium for strong bones.
The charity’s survey suggests that many young people seek dietary advice from bloggers and vloggers on the internet.
Although some of this advice can be good, the charity is concerned some people become too restrictive about what they eat.
A recent Food Standards Agency survey found that nearly half of 16-24 year olds said they had an intolerance to cow’s milk and dairy products, compared to just 8% of over-75s.
Yet only 24% had actually had their condition diagnosed by a doctor.
Prof Susan Lanham-New, head of nutritional sciences at the University of Surrey and clinical advisor to the National Osteoporosis Society, said: «Diet in early adulthood is so important because by the time we get into our late 20s it is too late to reverse the damage caused by poor diet and nutrient deficiencies and the opportunity to build strong bones has passed.»
What foods are rich in calcium?
Dairy products are the main source of calcium – for example, milk, cheese and yoghurt
Cow’s milk is the best source, but soya and almond milk may also contain calcium if they are fortified
Skimmed and semi-skimmed milk contain more calcium than full fat cow’s milk
Vegetables, nuts, seeds, fish and white flour products also contain calcium
Choose low-fat cheese and yoghurt to cut down on fat intake, although crisps and biscuits contain much more fat
For adults, 700mg of calcium per day is recommended but boys and girls between 11 and 18 need up to 1000mg.
However, a quarter of teenagers in the UK are thought to consume less than the minimum 400mg of calcium every day, dietary surveys suggest.
Recommended levels can be achieved by eating three portions of dairy a day, such as cereal with milk, a yoghurt and a small chunk of cheddar cheese, experts say.
Consuming foods rich in calcium and vitamin D, such as dairy products, green leafy vegetables, salmon, sardines, broccoli and baked beans, is particularly important before the age of 25, the osteoporosis charity said.
It is urging parents to talk to their children about their diet.
After the age of 50, half of all women and one in five men develop osteoporosis, a fragile bone condition that causes painful fractures of the hip, wrist and spine.
Smoking, lack of exercise and fizzy drinks high in acid are all detrimental to healthy bones.
A spokeswoman from the British Nutrition Foundation said: «While it’s not necessarily dangerous to cut out dairy from your diet it’s important to ensure you get enough calcium from other sources.
«Dairy tends to make the biggest contribution to our calcium intakes and so this needs to be replaced by other sources such as bread, cereal, canned fish, nuts, seeds and leafy green vegetables as well as choosing dairy alternatives that are fortified with calcium.»