I'm gonna be real, I do think Tristan could be genuine in his feeling like hes "not going to be able to share" just based on how Max acted in this episode. Setting aside any potential Max/Tristan attraction, Max spent this whole episode being uncomfortable around Tristan, butting heads with him, and asserting his authority over him in a way we haven't seen since episode 1, if ever. (He dressed Tristan down in front of the Captain and called him Nurse Silva has he ever done that??)
Then Max gives his whole speech about being traditional, wanting one woman, and children, and about how they need to be able to maintain their dynamics and therefore their hierarchy. I think Tristan quite frankly lost some trust he had gained in Max in this episode. Trust that he would be listened to, trust that they could meet on the same level, trust that Max trusted him in return.
Not to mention Avery actively used to his attraction to her to get what she wanted lol. He floats the idea of them making a good duo after Max's rejection and she won't even consider it. Then she flirts with him just to get him to agree to talk to Max. It was funny but you could see even then he looked a little put out by it. There's nothing wrong with her wanting both and feeling like monogamy isn't for her, but I do think Tristan took that as "I am not and will never be enough for her" and that hit right in his emotional issues.
I think it genuinely is difficult for him to see how they could be in a balanced relationship atp (poly-v OR triad). In his eyes, Max not only doesn't want to try something "non traditional" but the threesome has also brought his power over Tristan to the forefront, causing a breakdown in trust. Avery has, unintentionally, kinda made it seem like she isn't concerned for Tristan's feelings and/or just doesn't care enough about him. It makes sense that Tristan feels like he's not going to be respected or considered in the relationship!
I also think it's significant that Avery keeps talking about fun and not putting a label on it and just... Never talks about the emotional side of it which makes sense for her prior commitment issues but is something that both Max and Tristan explicitly want.
This also isn't to say it couldn't work out between them lol, I'm still rooting for them to at least be a V (tho it's be a triad in my heart). I think that jealousy and insecurity is normal sometimes in polyamory at the beginning. The key is to communicate and they aren't... Really doing that yet lol.