In a small pot over medium heat, warm the oil with the spices and orange zest. Cook for 1 minute. Add the olives and sliced fennel. Cook for 4 minutes. Remove from the heat. Add the orange and lemon juices, as well as the fennel fronds. Mix well. Serve hot or cold.