The pedestrians obey implicit social rules following behavioral patterns. These are not caused by deliberate actions but are rather a reaction to social forces that make individuals influence each other. These interactions in complex environments are not easy to model and, even when performing well in terms of prediction metrics, trajectory forecasting models might fail to grasp the underlying rules that dominate crowds. In real datasets, there are a lot of variables at play that contribute to the complexity of the problem, making such rules less evident and harder to model: the movement of an agent influences the trajectory of the others and since each agent can arbitrarily alter its motion, the problem is inherently multimodal, i.e., several predictions are needed to approximate the future. All these elements concur to make it extremely hard if not impossible to precisely assess and annotate cause-effect relationships in motion patterns of agents that conciliate their attempt to reach a destination with such aforementioned rules. To measure the capability of a model to perform correct predictions for this kind of patterns, we synthetically generated a dataset of interacting moving agents that obey a few simple social rules. This dataset, Synthetic Social Agents (SSA), is intended to be an oversimplification of real social behaviors, yet offers a challenging shift of attention from individual motion to social forces.