The study of slavery in the Americas generally assumes a basic racial hierarchy: Africans or those of African descent are usually the slaves, and white people usually the slaveholders. In this unique interdisciplinary work of historical archaeology, anthropologist Katherine Hayes draws on years of fieldwork on Shelter Island's Sylvester Manor to demonstrate how racial identity was constructed and lived before plantation slavery was racialized by the legal codification of races. Using the historic Sylvester Manor Plantation site turned archaeological dig as a case study, Hayes draws on artifacts and extensive archival material to present a rare picture of northern slavery on one of the North's first plantations. The Manor was built in the mid-17th century by British settler Nathaniel Sylvester, whose family owned Shelter Island until the early 18th century and whose descendants still reside in the Manor House. There, as Hayes demonstrates, white settlers, enslaved Africans, and Native Americans worked side by side. While each group played distinct roles on the Manor and in the larger plantation economy of which Shelter Island was part, their close collaboration and cohabitation was essential for the Sylvester family's economic and political power in the Atlantic Northeast. Through the lens of social memory and forgetting, this study addresses the significance of Sylvester Manor's plantation history to American attitudes about diversity, Indian land politics, slavery and Jim Crow, in tension with idealized visions of white colonial community.