Throwing knives are long range weapons that bounce of surfaces and have a slightly downwards path, but not as heavy as a grenade. They have high damage, though lack in attack speed. Their throwing arc can be somewhat tricky to get used to, but they benefit from never having to reload, technically.
Throwing knives work great with all kinds of weapons, but beneath a list of weapons and how great the throwing knives work:
Throwing knives are a great backup for snipers. As they provide a powerful short ranged weapon choice.
Throwing knives are great if you run out of ammo, which is likely with explosives, they are also useful is the rate of fire needed is higher than your primary weapon, as throwing knives have no need to reload.
Throwing knives have the same downwards path as grenade launchers, so they can be used as backup, when needing to reload.
The long reload times of machine guns mean that reloading in the heat of battle can be extremely dangerous. Switching to your secondary weapons instead is advised as it takes much less time and therefore minimizes the danger.
Throwing knives are a reserve for high damage magnums like no name's and .500's, and cover the lack of damage in for example, Taurus .44's. A good tactic is to equip a Hair Trigger to the magnum, and use the ability Full Auto. This allows the secondary Throwing Knives to be thrown at an increased rate.
Shotguns with minimal clip sizes are largely helped by quick secondary weapons. Switching to throwing knives instead of reloading is a strongly advised tactic.
As Elemental weapons have a generic lack of range, throwing knives help to balance this out with their ability to bounce off walls.