My issue with the airport is that it creates a barrier to the Toronto Islands from downtown. Imagine if Vancouver built a runway in the West End separating downtown Vancouver with Stanley Park. It's the same thing with Toronto.
Sure, you can take a ferry to the islands but it's not quite accessible. It doesn't feel like it's part of downtown and you can't just willy-nilly go there out of the blue. Without the airport as a physical and mental barrier and the area converted into parkland, the islands can start feeling part of downtown. Hanlan's Point Beach can be Toronto's Barceloneta beach, which will be more accessible than the Beaches. You can build a pedestrian/cycle/streetcar bridge across the Western Channel and have the 509 briefly make a few stops in that area.
Together with the Gardiner Expressway, this is just an example of poor urban planning and bad foresight.