You ask a very good question that is quite sensitive, and it functions on both halachik (Jewish legal) as well as on the psychological and interpersonal levels. Indeed, from a legal perspective we are prohibited from greeting mourners during their shiva period. Thus, one would think that wishing “L’shana Tova” would indeed be out of place to a mourner during that time.
However, there is no way that the bereaved family will be in the midst of their shiva mourning period on Rosh Hashanah. That is because shiva will, of necessity, either already be concluded or not yet be started. The Shulchan Aruch states (Yoreh Deah 399:1) that if a funeral has taken place before a major Jewish holiday, the holiday ends the shiva with its onset. So if the burial was even shortly before Rosh Hashanah, the moments of shiva before the holiday comprise the entirety of the shiva period. And if the death was before Rosh Hashanah but the funeral has to be delayed until after the holiday for some reason, then shiva will not yet have started. As such, there is no specific prohibition against greeting the mourner on Rosh Hashanah itself.
That said, there is still an interpersonal dimension that requires address. Rav Soloveitchik pointed out that in addition to the formal mourning legislated by Jewish law that includes sitting shiva and wearing a torn short and sitting on a low chair, there exists concurrently the “informal mourning” that naturally comes from a person’s upset mindset. Usually these two coincide, but when law cuts shiva short it doesn’t necessarily lessen in any way the deep pain and grief that the person is experiencing.
As a result, a standard holiday greeting may actually cause this person pain, making him or her feel as if everyone else is ignoring their loss. Saying nothing may also hurt, leaving the person feeling isolated and uncared for. Acknowledging the loss in your greeting may offer a respectful way to approach this. Perhaps most advisable would be to walk over to them and say “I can’t quite imagine how you are feeling right now, and I wish you a year of comfort, a better year than the one difficult one that just ended. How are you managing?”
I remember when I was a college student and my own rabbi's mother passed away suddenly right before Rosh Hashanah. On the holiday I approached him and asked if it was correct for me to still offer him a greeting of 'shanah tovah'. He said that it was especially important that he was offered such a wish, because it would be a difficult year for him as he was in mourning, but he hoped that it would still be a good year.
I recall that memory from my youth whenever there is a death in my community right before a holiday. I still make it a point to offer my wishes for a happy holiday, but also offer the caveat, that I know right now the family is not thinking about happiness or joy. Sometimes I say that I hope they have an enjoyable holiday (or festival), despite their feelings of sorrow over the loss of a loved one.
Generally speaking, it's always good to follow your heart. If your intuition tells you that it would be inappropriatae to wish a mourner a "Happy Holiday," then simply offer your condolences.
Simcha Paull Raphael, in his chapter in Jewish Pastoral Care, shares the following teaching, attributed to Abraham Joshua Heschel:
There are three ways to mourn: to weep, to be silent, and to sing.
The first way to mourn is to weep: even if our tears are for oruselves, for our ache of loneliness, for our pain of loss, they are still sacred, for they are the tears of love...
The second way to mourn is to be silent: to behold the mystery of love, to recall a shared moment, to remember a word or a glance, or simply at some unexpected momemnt, to miss someone very much and wish that he or she could be here...
The third way to mourn is to sing: to sing a hymn to life, a life that still abounds in sights and sounds and vivid colors; to sing the song our beloved no longer has a chance to sing.
In immediate loss, any three of these forms of mourning might be happening for our friends, and certainly proximity to the holidays add a level of intensity to the loss.
So the question becomes: how do we greet people at the holidays when we know they're suffering a loss?
The answer is to acknowledge and give space to their mourning--their weeping, their silence and their song. In greeting, we create the opportunity for the bereaved to share their grief in the manner that fits the moment.
This means that, instead of merely offering a "L'shana tova" or "Chag Sameach" we may want to stop and offer ourselves to them. Express our own sorrow for their loss, check in and ask how they're doing. Offer to spend more time with them. In doing so, we are doing more than simply exchanging greetings; we're creating a community of care for the person. They may simply nod and smile (or not), or share their brokenness through tears, or speak about how much dad loved the music of the high holidays and that will always be with them. The point is, you gave them the opportunity to weep, to be silent, or to sing.